Data Management
This section desribes the processes and strategies that are used for collecting, storing, managing, protecting, and maintaining data in CCC, with particular focus on persistence, backup, and restore mechanisms. Persistence ensures that data is not lost when the CCC container is shut down, while backup and restore are the key components of data protection and disaster recovery strategies. These processes contribute towards ensuring the availability, integrity, and security of CCC data, and form the core of CCC data management.
Database Reset
The database reset function of CCC serves to reset the CCC application by dropping the existing database and starting it with a fresh schema. This process effectively reverts the application to its original configuration, eliminating any previously stored database. It is particularly useful when there is a need to clear existing database and restore the application to its initial state. Data reset is commonly employed for troubleshooting, testing, or setting up a fresh database for specific use cases. To implement the database reset process, follow the steps below:
Go to the machine where the CCC application is running.
Execute the following command, depending on the method you've used for CCC installation:
Podman
podman exec ccc bash -c 'drop-db <database_password>'
Kubernetes/Helm
kubectl exec <ccc-pod-name> -- bash -c 'drop-db <database_password>'
Replace <ccc-pod-name> with the name of the pod running the CCC application and <database_password> with the appropriate password required to drop the database.
In case your database password includes any special characters, it is essential to insert a backslash before each special character to ensure correct execution of the command.
If multiple Worker nodes are running in your Kubernetes/Helm cluster, executing the database reset command on one Worker node can render the other nodes unusable. To prevent any potential issues, we strongly recommend keeping only one instance of CCC active while executing the database reset command. Once the database reset is complete, you can scale up to more nodes as needed. To ensure a smooth data refresh process, please execute the following command from the Master node to scale down to only one instance of CCC:
kubectl scale --replicas=1 deployment ccc-deployment
Please exercise caution when using the data reset function, as it will permanently remove all existing data from the CCC application. Ensure that you have a valid backup of critical data before proceeding with this operation.
Persistence
The persistence mechanisms supported by CCC include bind mount and volume mount.
Bind Mount
The default persistence mechanism used by CCC is bind mount, wherein the CCC container mounts a file or directory from the host machine's file system into the container's file system. This allows the container to access and modify the data in the file or directory as if it were part of its own file system. Any changes made to the data by the container are also reflected on the host machine's file system, thus persisting the changes even after the container is stopped or deleted.
Podman
For users who have installed CCC with Podman, the /var/lib/postgresql directory of the CCC container is mapped to the ccc_dist_folder/podman/pgdata directory on the host machine. The entire CCC database is stored inside the pgdata directory. When the podman-compose up command is used to initialize the CCC container, the container obtains the mapping from the podman-compose.yml file and begins persisting data according to the defined specifications.
You can change the directory that has been mapped on the host machine by modifying the podman-compose.yml file.
Kubernetes
For users who have installed CCC with Kubernetes, the /var/lib/postgresql directory of the CCC container is mapped to /home/ccc/pgdata directory on the host machine. The entire CCC database is stored inside the pgdata directory. When the CCC container is initialized, it obtains the mapping from the postgres-data.yaml file and begins persisting data according to the defined specifications.
You can change the directory that has been mapped on the host machine by modifying the postgres-data.yaml file.
Volume Mount
To change the CCC persistence mechanism to volume mount that enables data to be persisted and shared between containers, or between a container and the host machine, you need to make the following changes, depending on the orchestration mechanism that has been used while installing CCC:
Podman
If you’ve installed CCC using Podman and want to change the persistence mechanism to volume mount, use the steps outlined below:
Open the podman-compose.yml file in the podman directory.
Go to the volumes section.
Replace -./pgdata:/var/lib/postgresql with -ccc_pgdata:/var/lib/postgresql:Z,U.
Add a new Volumes section in podman-compose.yml file, with the following details:
volumes:
ccc_pgdata:
Ensure that the updated podman-compose.yml file appears as follows after you've made these changes:
The Podman volume data gets stored at:
/home/user/.local/share/containers/storage/volumes/podman_volume.
Therefore, after changing the persistence mechanism to volume, the data directory will be at:
/home/user/.local/share/containers/storage/volumes/podman_ccc_pgdata.
Kubernetes
If you’ve installed CCC using Kubernetes and want to change the persistence mechanism to volume mount, use the steps outlined below:
Open the deployment.yaml file in the Kubernetes directory.
Scroll to the volumes section.
Replace the existing content of the volumes section with the following content:
- name: lunalogs
emptyDir: {}
- name: packages
emptyDir: {}
- name: ccclogs
emptyDir: {}
- name: pgdata
emptyDir: {}
- name: certificates
hostPath:
path: /home/ccc/ccc-certs/
type: Directory
It is not advisable to use the volume mount mechanism to persist data in Kubernetes. Kubernetes stores data at the path /var/lib/kubelet/pods/pod-id/volumes/kubernetes.io~empty-dir inside the worker node and if the pod or deployment gets deleted, any data that is persisted using this approach will be lost permanently.
Backup and Restore
The approach you need to use to backup and restore CCC data depends on whether you are using an internal database or an external database.
Internal Database: Backup and Restore
External Database: Backup and Restore
Internal Database: Backup and Restore
If CCC is using an internal database, it utilizes the postgresql database running within the same container. To ensure data persistence, a bind mount mechanism is employed, whereby the /var/lib/postgresql directory of the CCC container is linked to the host machine. To backup and restore data, you can use either of the following two methods:
Method 1: Backup and restore using pg_dump
Method 2: Backup and restore using pgdata
Method 1: Backup and restore using pg_dump
This method involves creating a pg_dump backup file and then restoring it.
Backup
The steps involved in performing a backup are as follows:
Create a pg_dump file by executing below command:
Podman
podman exec ccc bash -c "/usr/pgsql-14/bin/pg_dump 'host=localhost port=5432 dbname=lunadirectordb user=lunadirector password=temp1234' > postgres.sql"
Kubernetes
kubectl exec -it <podname> -- bash -c "/usr/pgsql-14/bin/pg_dump 'host=localhost port=5432 dbname=lunadirectordb user=lunadirector password=temp1234' > postgres.sql"
Copy the pg_dump file from container to host machine to make it available for future.
Podman
podman cp ccc:/usr/safenet/ccc/postgres.sql .
Kubernetes
kubectl cp <podname>:/usr/safenet/ccc/postgres.sql .
Restore
This method involves restoring the pg_dump backup file. The steps involved are as follows:
Copy the backup pg_dump file into the container directory /usr/safenet/ccc/.
Podman
podman cp postgres.sql ccc:/usr/safenet/ccc/
Kubernetes
kubectl cp postgres.sql <podname>:/usr/safenet/ccc/
Execute the below command to restore the backup data:
Podman
podman exec ccc bash -c "cd scripts && sh dbmig.sh '<ccc_admin_password>'"
Kubernetes
kubectl exec <podname> bash -c "cd scripts && sh dbmig.sh '<ccc_admin_password>'"
If the CCC admin password contains special characters, we need to enter those with escape. For example, if the CCC admin password is "1q@W3e$R", we need to enter it as "1q\@W3e\$R".
The name of the pg_dump file should be "postgres.sql".
Method 2: Backup and Restore using pgdata
This method involves making a backup of pgdata directory at a safe location and restoring it back.
Backup
To create a backup, copy the pgdata directory and place it at a safe location as a backup.
Restore
To restore the database from the backup file, perform these steps:
Go to the podman directory where the podman-compose.yml file is present and remove the running CCC container by using the following command:
podman-compose down
Remove the current pgdata directory in the podman directory.
Copy the backup pgdata directory and paste it into the podman directory.
Go to podman directory, update the ccc_config.env file if needed, and then execute the following command:
podman-compose up -d
External Database: Backup and Restore
If you are utilizing an external database and need to perform data backup and restore, refer to the procedure outlined on the official postgres website:
https://www.postgresql.org/docs/current/backup.html